HiveMQ Enterprise Extension for MongoDB

MongoDB is an open-source, document-oriented database system. MongoDB is designed for scalability, high availability, and performance which makes it suitable for various types of applications, including big data, analytics, and IoT.

Unlike traditional relational databases that use tables, rows, and columns to store data, MongoDB uses a JSON-like format called BSON (Binary JSON) to store documents. These documents are organized in collections, that are analogous to the tables in a relational database.

Features

Forward MQTT messages from IoT devices to one or more MongoDB databases via your HiveMQ broker.
Convert MQTT messages into MongoDB documents with a flexible templating system.

Requirements

A running HiveMQ Professional or Enterprise Edition installation, version 4.17 or higher.
A running MongoDB instance, version 4.4 or higher is recommended.
For production use, a valid HiveMQ Enterprise Extension for MongoDB license.

If you do not provide a valid license, HiveMQ automatically uses a free trial license. Trial licenses for HiveMQ Enterprise Extensions are valid for 5 hours. For more license information or to request an extended evaluation license, contact HiveMQ sales.

Installation

Place your HiveMQ Enterprise Extension for MongoDB license file (.elic) in the license folder of your HiveMQ installation. (Skip this step if you are using a trial version of the extension).

└─ <HiveMQ folder>
    ├─ bin
    ├─ conf
    ├─ data
    ├─ extensions
    │   ├─ hivemq-mongodb-extension
    │   └─ ...
    ├─ license
    ├─ log
    └─ ...

Before you enable the extension, you need to configure the extension to match your individual MongoDB setup.
For your convenience, we provide an example configuration conf/examples/config.xml that you can copy and modify as desired.
The included config.xsd file outlines the schema and elements that can be used in the XML configuration.
Your completed configuration file must be named config.xml and located in HIVEMQ_HOME/extensions/hivemq-mongodb-extension/conf/config.xml.
For detailed information on configuration options, see Configuration.
To enable the HiveMQ Enterprise Extension for MongoDB, locate the hivemq-mongodb-extension folder in the extensions directory of your HiveMQ installation and remove the DISABLED file (if present).

To function properly, the HiveMQ Enterprise Extension for MongoDB must be installed on all HiveMQ broker nodes in your HiveMQ cluster and the configuration file on each node must be identical.

Configuration

The extension configuration is divided into two sections:

MongoDBs: Provides information about the MongoDB instances to which your HiveMQ broker connects.
MQTT to MongoDB Routes: Defines how MQTT messages are sent from your HiveMQ broker to the configured MongoDB instances.

Extension Configuration File

The config.xml file for your MongoDB extension must be located in the hivemq-mongodb-extension/conf folder within the extensions folder of your HiveMQ installation.

The extension uses a simple but powerful XML-based configuration.

The conf/examples/config.xml file is a configuration example that has all the parameters you need to send MQTT messages from your HiveMQ MQTT broker to MongoDB.

If you copy and reuse the conf/examples/config.xml file, be sure to move the file to /conf/config.xml before you enable your extension. For more information, see Installation.

Example MongoDB extension configuration

<hivemq-mongodb-extension xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                          xsi:noNamespaceSchemaLocation="config.xsd">
    <mongodbs>
        <mongodb>
            <id>my-mongodb-id</id>
            <connection>
                <host>my.host.com</host>
                <port>27017</port>
            </connection>
        </mongodb>
    </mongodbs>

    <mqtt-to-mongodb-routes>
        <mqtt-to-mongodb-route>
            <id>my-mqtt-to-mongodb-route</id>
            <mongodb-id>my-mongodb-id</mongodb-id>
            <mqtt-topic-filters>
                <mqtt-topic-filter>#</mqtt-topic-filter>
            </mqtt-topic-filters>
            <database>my-database</database>
            <collection>my-collection</collection>
            <processor>
                <document-template>path/to/my/document-template.json</document-template>
            </processor>
        </mqtt-to-mongodb-route>
    </mqtt-to-mongodb-routes>
</hivemq-mongodb-extension>

MongoDB Databases Configuration

The <mongodbs> section of your configuration lists the MongoDB databases to which configured MQTT messages are routed.

You can define as many <mongodb> tags as your use case requires.

Example MongoDB database connection configuration

<mongodb>
    <id>my-mongodb-id</id>
    <connection>
        <host>my.host.com</host>
        <port>27017</port>
    </connection>
</mongodb>

Example MongoDB database connection with basic authentication

<mongodb>
    <id>my-mongodb-id</id>
    <connection>
        <host>my.host.com</host>
        <port>27017</port>
        <authentication>
            <basic>
                <username>example-user</username>
                <password>example-password</password>
            </basic>
        </authentication>
    </connection>
</mongodb>

Basic authentication expects the user credentials to be in the 'admin' authentication database (default). The authentication mechanism for MongoDB version 4.0 or higher is SCRAM-SHA-256.

Example MongoDB connection string

<mongodbs>
    <mongodb>
        <id>my-mongodb-id</id>
        <connection-string>mongodb://example-user:example-password@my-host.com:27017/</connection-string>
    </mongodb>
</mongodbs>

Example MongoDB connection string using a DNS Seed List

<mongodbs>
    <mongodb>
        <id>my-mongodb-id</id>
        <connection-string>mongodb+srv://example-user:example-password@my.dns-srv-record.com/</connection-string>
    </mongodb>
</mongodbs>

Example MongoDB TLS configuration

<mongodb>
    <id>my-mongodb-id</id>
    <connection-string>mongodb://example-user:example-password@my-host.com:27017/</connection-string>
    <tls>
        <keystore>
            <path>/keystore.jks</path>
            <private-key-password>my-private-key-password</private-key-password>
            <password>my-keystore-password</password>
        </keystore>
        <truststore>
            <path>/truststore.jks</path>
            <password>my-truststore-password</password>
        </truststore>
        <verifyHostname>false</verifyHostname>
    </tls>
</mongodb>

You can use Environment variables to map properties such as passwords or connection strings.

Table 1. MongoDB parameters
Parameter	Required	Type	Description
`id`		ID	The ID of the MongoDB instance. This string can only contain lowercase alphanumeric characters, dashes, and underscores.
`connection`		Complex	The connection information for the MongoDB instance. NOTE: Your configuration must contain either a `connection` parameter that lists a `host` and `port` or a `connection-string` parameter that contains the appropriate connection string. `host`: The host name of the MongoDB instance. `port`: The port of the MongoDB instance.
`connection-string`	-	String	A connection string for the MongoDB instance.
`authentication`	-	Complex	Defines the type of authentication the extension uses for the MongoDB connection. By default, no values are set. `basic`: Optional authentication configuration for the connection to MongoDB that uses SCRAM authentication. `username`: The username required to establish the connection to the MongoDB instance. `password`: The password required to establish the connection to the MongoDB instance.
`tls`	-	Complex	Optional TLS configuration to establish a secure connection to MongoDB. `enabled`: Optional setting that defines whether the selected TLS configuration is used to encrypt communication between the extension and MongoDB. The default setting is `true`. To disable TLS, set the value to `false`. `keystore`: When TLS is enabled, a keystore can be configured to contain the certificate of the MongoDB instance. `path`: The path to the PKCS#12 (.p12/.pfx) or JKS (.jks) keystore where client certificates are stored. `private-key-password`: The password for the private key. `password`: The password for the keystore. `truststore`: Optional configuration of a truststore that contains the server certificate or the certificate of the certificate authority (CA) that issued the server certificate. `path`: The path to the PKCS#12 (.p12/.pfx) or JKS (.jks) truststore where trusted certificates are stored. `password`: The password for the truststore. `verify-hostname`: Optional setting that defines whether the extension checks the identity of the remote server. The default setting is `true`. To skip the verification, set the value to `false`.

MQTT to MongoDB Routes

The <mqtt-to-mongodb-routes> section of your extension configuration defines how MQTT messages are sent from the HiveMQ broker to MongoDB.

You can define as many <mqtt-to-mongodb-route> tags as your use case requires.

Example MQTT to MongoDB route

<mqtt-to-mongodb-routes>
    <mqtt-to-mongodb-route>
        <id>my-mqtt-to-mongodb-route</id>
        <mongodb-id>my-mongodb-id</mongodb-id>
        <mqtt-topic-filters>
            <mqtt-topic-filter>myhome/groundfloor/+/temperature</mqtt-topic-filter>
            <mqtt-topic-filter>myhome/firstfloor/#</mqtt-topic-filter>
            <mqtt-topic-filter>myhome/secondfloor/bedroom/temperature</mqtt-topic-filter>
        </mqtt-topic-filters>
        <database>my-database</database>
        <collection>my-collection</collection>
        <processor>
            <document-template>path/to/my/document-template.json</document-template>
        </processor>
    </mqtt-to-mongodb-route>
</mqtt-to-mongodb-routes>

Table 2. MongoDB parameters
Parameter	Required	Type	Description
`id`		ID	The ID of the `mqtt-to-mongodb-route`. This string can only contain lowercase alphanumeric characters, dashes, and underscores.
`mongodb-id`		IDREF	The ID of the MongoDB instance.
`enabled`	-	Boolean	Optional setting that defines whether the selected `mqtt-to-mongodb-route` is enabled or disabled. The default setting is `true`. To disable the route, set to `false`.
`mqtt-topic-filters`		Complex	A list of one or more MQTT topic filters. `mqtt-topic-filter`: The source MQTT topic filters to select which MQTT messages are routed to the MongoDB database. You can define as many individual `<mqtt-topic-filter>` tags as your use case requires.
`database`		String	The MongoDB database the MQTT messages are routed to.
`collection`		String	The MongoDB collection the MQTT messages are routed to.
`processor`		Complex	Defines how MQTT messages are converted into MongoDB documents. `document-template`: The path to a JSON document template for the MongoDB documents. The path can be an absolute path or relative to the home folder of your HiveMQ extension for MongoDB.

Document Templates

Document templates are JSON files that help you convert MQTT messages into MongoDB documents. These templates allow you to set placeholders that your HiveMQ extension later fills with the properties of the MQTT PUBLISH.

Placeholders start with ${ and end with }.

Example document template

{
  "topic": "${mqtt-topic}",
  "payload_utf8": "${mqtt-payload-utf8}",
  "qos": "${mqtt-qos}",
  "retain": ${mqtt-retain},
  "packet_id": ${mqtt-packet-id},
  "payload_format_indicator": "${mqtt-payload-format-indicator}",
  "response_topic": "${mqtt-response-topic}",
  "correlation_data_utf8": "${mqtt-correlation-data-utf8}",
  "user_properties": ${mqtt-user-properties-json},
  "my_user_property": "${mqtt-user-properties$myUserPropertyName}",
  "arrival_timestamp": ${timestamp-ms}
}

Example document template for JSON payloads

{
  "packet_id": ${mqtt-packet-id},
  "payload_utf8": ${mqtt-payload-utf8}
}

If you expect JSON payloads, check that the ${mqtt-payload-utf8} placeholder is not surrounded with double quotes to ensure that MongoDB interprets the payload content as JSON rather than String.

Example document template with a number placeholder and ISODate() function

{
  "packet_id": ${mqtt-packet-id},
  "arrival_timestamp": ISODate("${timestamp-iso-8601}")
}

Although this example is not a valid JSON, the example works since the extension replaces placeholders such as mqtt-packet-id. In the example, the placeholder is replaced with a number and MongoDB interprets the ISODate() function.

Your HiveMQ extension automatically checks for the presence of unknown placeholders in your document template. If an unknown placeholder is present, HiveMQ logs an error and the HiveMQ Extension for MongoDB does not start.

The following table lists all placeholders the HiveMQ MongoDB extension recognizes:

Table 3. Available placeholders
Name	Optional properties	Description
`mqtt-topic`	-	The topic of the MQTT PUBLISH.
`mqtt-payload-utf8`		The payload of the MQTT PUBLISH as a UTF-8 string.
`mqtt-payload-base64`		The payload of the MQTT PUBLISH as a Base64 string.
`mqtt-qos`	-	The QoS level of the MQTT PUBLISH (AT_MOST_ONCE, AT_LEAST_ONCE, EXACTLY_ONCE).
`mqtt-retain`	-	The retain message flag of the MQTT PUBLISH.
`mqtt-packet-id`	-	The packet id of the MQTT PUBLISH.
`mqtt-payload-format-indicator`		The payload format indicator of the MQTT PUBLISH.
`mqtt-response-topic`		The response topic of the MQTT PUBLISH.
`mqtt-correlation-data-utf8`		The correlation data of the MQTT PUBLISH as a UTF-8 string.
`mqtt-correlation-data-base64`		The correlation data of the MQTT PUBLISH as a Base64 string.
`mqtt-user-properties-json`		The user properties of the MQTT PUBLISH as a JSON array string.
`mqtt-user-properties$property-name`		The value of the user property of the MQTT PUBLISH with the matching property name. Example: `mqtt-user-properties$myUserPropertyName` will bind the value of the first user property with the name `myUserPropertyName`.
`timestamp-ms`	-	The arrival timestamp of the PUBLISH message represented as a UNIX timestamp value in milliseconds. The timestamp of an incoming PUBLISH message records the moment the message arrived at the broker. The timestamp of a message created via the Extension SDK records the moment the message passed to the PublishService. The timestamp of a Will Publish message sent to subscribers records the moment the delivery of the message started.
`timestamp-iso-8601`	-	The same information as `timestamp-ms` represented as an ISO 8601 string.

Some properties in an MQTT PUBLISH message are optional. The number of placeholders the MongoDB extension fills varies based on the properties that are present in the MQTT PUBLISH message. Optional properties in the document template are replaced with an empty string if the property is not present in the MQTT PUBLISH message.

Environment variables

HiveMQ offers placeholders that can be replaced with the content of environment variables when the configuration file is read. For many use cases, it can be beneficial or necessary to use environment variables to configure items such as ports and bind addresses on the system on which you run HiveMQ. For example, when you run HiveMQ in a containerized environment.

You can use ${YOUR_ENVVAR_NAME} in the config.xml file. HiveMQ replaces the placeholder with the value of the specified environment variable during startup.

Example to set an environment variable

export MY_CONNECTION_STRING=mongodb://example-user:example-password@my-host.com:27017/

Example use of the environment variable in the configuration

<hivemq-mongodb-extension xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                          xsi:noNamespaceSchemaLocation="config.xsd">
    <mongodbs>
        <mongodb>
            <id>my-mongodb-id</id>
            <connection-string>${MY_CONNECTION_STRING}</connection-string>
        </mongodb>
    </mongodbs>
</hivemq-mongodb-extension>

Result of the example configuration in HiveMQ

<hivemq-mongodb-extension xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                          xsi:noNamespaceSchemaLocation="config.xsd">
    <mongodbs>
        <mongodb>
            <id>my-mongodb-id</id>
            <connection-string>mongodb://example-user:example-password@my-host.com:27017/</connection-string>
        </mongodb>
    </mongodbs>
</hivemq-mongodb-extension>

Make sure that HiveMQ is started in the same context as your environment variables are set, otherwise HiveMQ will not be able to access them.